Drop the SHARE_GPU=1 escape hatch from run.sh. Colocating ranks on a
single GPU silently bypasses the symmetric-heap / XGMI path and reports
false-positive PASSes — exactly what the test exists to validate.
Replace with a precondition check that exits non-zero when fewer GPUs
are visible than ranks were requested. Validated on rad-mi325x-1
(8x MI325X) at WORLD_SIZE=2,4,8.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Introduce an AIR primitive for the symmetric-heap pointer rebase, in
preparation for the kernel-driven producer/consumer redesign per
@mawad-amd's review feedback on PR Xilinx#1577.

  %peer = air.translate %src, %from, %to, %bases : memref<NxT, A>, !llvm.ptr

Signature:
- $source: memref on $from_rank's symmetric heap
- $from_rank, $to_rank: index-typed rank ids
- $heap_bases: !llvm.ptr to the per-rank base table from mgpuGetHeapBases()
- result: same memref type, addressing $to_rank's slice of the same
  collective allocation

The op is Pure and folds when from_rank == to_rank (statically equal SSA
values or matching constant attrs). Naming follows IRIS's `__translate`.

Lowering pass `air-translate-to-llvm` expands each op to the
peer-VA arithmetic plus a freshly-built LLVM memref descriptor:

  byte_diff = ptrtoint(bases[to]) - ptrtoint(bases[from])
  peer_aligned_ptr = src_aligned_ptr + byte_diff   (i8 GEP)
  build descriptor { peer_ptr, peer_ptr, 0, sizes, strides }
  unrealized_conversion_cast back to result memref type

The expansion is pure arithmetic (arith + memref + llvm dialect), no
runtime calls — therefore valid both at host scope and inside `gpu.func`,
provided heap_bases is threaded as a kernel argument.

Tests:
- mlir/test/Dialect/AIR/air_translate.mlir: parser/printer + folder
- mlir/test/Conversion/AIRToROCDL/air_translate_to_llvm.mlir: lowering
  shape on 1D, 2D-addrspace, gpu.func body, and no-op cases

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Per @mawad-amd's review feedback on PR Xilinx#1577: replace the host-orchestrated
mgpuMemcpy reference test with a kernel-driven producer/consumer pair.
Cross-rank data movement is now performed by GPU compute units issuing
loads/stores directly into peer HBM over XGMI, not by the HIP copy engine.

Changes:

- air_sym_handwritten.mlir is rewritten as one gpu.module with two
  gpu.func kernels:
    * producer (rank 0): each thread writes 42.0 into rank 1's `data`
      via memref.store on a peer memref produced by air.translate.
      Lane 0 of each warp signals the per-warp flag with a release
      atomicrmw on rank 1's `flags`.
    * consumer (rank 1): lane 0 of each warp spins on its flag with an
      acquire atomic load until producer signals; gpu.barrier then
      releases all 64 lanes to read their data slot and copy it into
      a verify buffer. Host D2H reads verify_buf and checks 42.0.
  The host driver (func.func @main) initializes the symmetric heap,
  copies heap_bases into a device-resident buffer (workaround for the
  fact that mgpuGetHeapBases returns a host pointer), and dispatches
  the producer or consumer kernel based on rank.

- run.sh adds the GPU compilation chain (rocdl-attach-target,
  convert-gpu-to-rocdl, gpu-module-to-binary, gpu-async-region,
  gpu-to-llvm) before mlir-runner.

- run.sh sets HIP_VISIBLE_DEVICES=$i + LOCAL_RANK=0 per process so each
  rank sees only its own GPU as device 0. This eliminates the
  device-binding ambiguity between airgpu's hipSetDevice and MLIR's
  built-in gpu.launch_func handling that would otherwise cause rank N>0
  to fail with hipErrorInvalidDevice when launching kernels.

Validated on rad-mi325x-1 (8x MI325X, ROCm 7.1.1):
  W=2: rank 1 (consumer): cross-rank kernel write PASS (verify[0]=42.0)
  W=4: ALL 4 RANKS PASSED (rank 0/1 active, ranks 2-3 idle)
  W=8: ALL 8 RANKS PASSED (rank 0/1 active, ranks 2-7 idle)

This is the first time GPU compute units (not the HIP copy engine)
have been observed driving cross-rank data movement over XGMI in this
stack.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Two CI fixes:

1. air_translate_to_llvm.mlir: add `// REQUIRES: gpu`. The pass
   `--air-translate-to-llvm` is only registered when AIR_ENABLE_GPU=ON
   (it lives in the gpu-only conversion-pass set). Without the gate
   the test fails on non-GPU builds with
     air-opt: Unknown command line argument '--air-translate-to-llvm'
   This matches the pattern already used by the sibling tests
   air_to_rocdl.mlir and air_gpu_outlining.mlir.

2. AIRTranslateToLLVMPass.{h,cpp}: clang-format-17 reflow. The header
   banner had a too-long filename which clang-format wrapped into a
   broken two-line banner ("//===- ...PASS.h ----*- C++\n//-*-===//"),
   and a few function calls in the .cpp wanted slightly different
   wrapping. Match the surrounding header-banner convention (80 cols
   wide) and accept the .cpp reflow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

New conversion pass that replaces each `air.rank` op by inlining its body
in place, with rank IDs computed at runtime via `mgpuGetRank()` and
delinearized into the rank's N-D iteration space. Replaces
`air-rank-to-launch` for the GPU pipeline (which serialized ranks via
scf.for — a placeholder for single-process execution).

After this pass each process executes the entire `air.rank` body once,
with its rank id resolved dynamically from the runtime. Heap lifecycle
(`mgpuSymmetricHeapInit` / `mgpuSymmetricHeapDestroy`) is bracketed
around the parent function once per function (not per rank).

- `mlir/include/air/Conversion/AIRRankToMgpuPass.h` — public header
- `mlir/include/air/Conversion/GPUPasses.td` — `air-rank-to-mgpu` def
  with `heap-size` option (default 256 MB)
- `mlir/include/air/Conversion/GPUPassDetail.h` — `GEN_PASS_DEF_AIRRANKTOMGPU`
- `mlir/lib/Conversion/AIRRankToMgpuPass.cpp` — pass implementation
- `mlir/lib/Conversion/CMakeLists.txt`, `Passes.cpp` — registration
- `mlir/test/Conversion/AIRRankToMgpu/rank_to_mgpu.mlir` — FileCheck
  unit tests (10 cases; see Test plan below)
- `test/gpu/symmetric_heap_dma/air_sym_with_rank.mlir` — high-level
  air.rank-based equivalent of the Phase 2 hand-written reference
- `test/gpu/symmetric_heap_dma/run.sh` — `INPUT=rank|handwritten`
  selector to run either form through the same multi-process driver

FileCheck unit tests cover:
- 1D / 2D rank delinearization (remsi/divsi)
- Default + custom heap-size option
- Async form (token replacement via wait_all)
- Async dependencies (blocking wait_all insertion)
- Multiple `air.rank` ops per function (init/destroy emitted once)
- Multiple `func.return` paths (destroy before each)
- Kernel operand mapping (block args replaced by SSA operands)
- Idempotent extern decls across multiple functions
- No-op when no `air.rank` is present (audit-found bug fixed: pass was
  unconditionally inserting decls)

End-to-end: rad-mi300a-sh5-1, SHARE_GPU=1, 2 ranks, INPUT=rank — both
ranks PASS the cross-rank read.

Caveat: same SHARE_GPU=1 single-physical-GPU caveat as Phase 2. True
multi-GPU re-validation is needed before declaring multi-GPU production-
ready (blocked on ROCm-side work).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

New conversion pass that replaces `memref.alloc` carrying the unit
attribute `air.symmetric` with a call to `mgpuSymmetricAlloc(size, stream)`.
The returned `!llvm.ptr` is wrapped in an LLVM memref descriptor (struct)
and projected back to the original memref type via
`builtin.unrealized_conversion_cast` so downstream uses keep working
through the standard `convert-to-llvm` pipeline.

`memref.dealloc` ops whose operand traces back (through the cast) to a
symmetric alloc are rewritten to `mgpuSymmetricFree`.

The pass is a no-op when no `air.symmetric` allocations are present.

- `mlir/include/air/Conversion/AIRSymmetricAllocToMgpuPass.h` — header
- `mlir/include/air/Conversion/GPUPasses.td` — `air-symmetric-alloc-to-mgpu` def
- `mlir/include/air/Conversion/GPUPassDetail.h` — `GEN_PASS_DEF_AIRSYMMETRICALLOCTOMGPU`
- `mlir/lib/Conversion/AIRSymmetricAllocToMgpuPass.cpp` — implementation
- `mlir/lib/Conversion/{CMakeLists.txt,Passes.cpp}` — registration
- `mlir/test/Conversion/AIRSymmetricAllocToMgpu/symmetric_alloc.mlir` — FileCheck
- `test/gpu/symmetric_heap_dma/air_sym_with_alloc.mlir` — high-level e2e
  using `memref.alloc {air.symmetric}` (Phase 3 + Phase 4 chained)
- `test/gpu/symmetric_heap_dma/run.sh` — `INPUT=alloc` selector

FileCheck unit tests:
- 1D alloc + dealloc shape (size, descriptor, cast, free)
- 2D alloc with row-major strides in descriptor
- Element type byte-size: f32 (4B), f64 (8B), i32 (4B)
- Multiple symmetric allocs share one decl pair
- Pass is a no-op for non-symmetric allocs
- Pass is a no-op when there are zero symmetric allocs

End-to-end on rad-mi300a-sh5-1 (SHARE_GPU=1, 2 ranks):
- INPUT=handwritten — PASS (Phase 2 baseline)
- INPUT=rank — PASS (Phase 3)
- INPUT=alloc — PASS (Phase 4: chained Phase 4 + Phase 3 lowering)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

New conversion pass that lowers `air.dma_memcpy_nd` ops carrying a
`src_rank` or `dst_rank` integer attribute (added in Phase 1) to host-side
`mgpuMemcpy` calls with peer-VA addressing through `mgpuGetHeapBases()`.

The peer pointer is computed at runtime as:
  peer_ptr = bases[peer_rank] + (local_ptr - bases[my_rank])

where `local_ptr` is extracted from the local-side memref via
`memref.extract_aligned_pointer_as_index` and `local_base = bases[my_rank]`
gives this rank's symmetric heap base.

- Both `src` and `dst` memrefs must be in `memory_space=0` (L3/global)
- The op must be at host scope (not inside a `gpu.launch` or `gpu.func`)
- "Entire memref" form only — no explicit `[offsets][sizes][strides]`
- Only one of `src_rank` / `dst_rank` may be set per op

These restrictions match the hand-written reference's Phase 2 pattern. They
can be relaxed in follow-up work.

- `mlir/include/air/Conversion/AIRCrossRankDmaToMgpuPass.h` — header
- `mlir/include/air/Conversion/GPUPasses.td` — `air-cross-rank-dma-to-mgpu` def
- `mlir/include/air/Conversion/GPUPassDetail.h` — `GEN_PASS_DEF_AIRCROSSRANKDMATOMGPU`
- `mlir/lib/Conversion/AIRCrossRankDmaToMgpuPass.cpp` — implementation
- `mlir/lib/Conversion/{CMakeLists.txt,Passes.cpp}` — registration
- `mlir/test/Conversion/AIRCrossRankDmaToMgpu/cross_rank_dma.mlir` — FileCheck
- `test/gpu/symmetric_heap_dma/air_sym_with_dma.mlir` — high-level e2e
  combining Phase 1 attrs + Phase 3 + Phase 4 + Phase 5 lowering
- `test/gpu/symmetric_heap_dma/run.sh` — adds `INPUT=dma` selector

FileCheck unit tests cover:
- src_rank lowering shape (size, ptr extraction, bases, GEP, ptrtoint, subi,
  byte-stride GEP, mgpuMemcpy)
- dst_rank lowering (peer pointer becomes dst arg)
- 2D memref byte size
- f64 element type byte size
- Multiple cross-rank DMAs share extern decls
- Pass is a no-op for non-cross-rank DMAs

End-to-end on rad-mi300a-sh5-1 (SHARE_GPU=1, 2 ranks):
- INPUT=handwritten — PASS (Phase 2 baseline)
- INPUT=rank — PASS (Phase 3)
- INPUT=alloc — PASS (Phase 4)
- INPUT=dma — PASS (Phase 5: chains Phase 5 -> Phase 4 -> Phase 3)
  Both ranks read rank 0's symmetric src_buf via cross-rank DMA into their
  own dst_buf; verification reads back 1.0.

Same SHARE_GPU=1 single-physical-GPU caveat as Xilinx#1577 / Xilinx#1578 / Xilinx#1579 —
true multi-GPU re-validation is needed before declaring multi-GPU
production-ready.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

New conversion pass that lowers `air.channel` ops of
`channel_type = "gpu_symmetric_heap"` plus their put/get pair to
host-side `mgpuMemcpy` calls with peer-VA addressing through
`mgpuGetHeapBases()`, with `mgpuBarrier`-based synchronization.

Per channel:
  - put becomes `mgpuBarrier()` (publish — the data is already in the
    symmetric heap via the put's `air.symmetric` source memref)
  - get becomes `mgpuBarrier()` followed by `mgpuMemcpy(dst, peer_va(src), sz)`
    where the peer rank is the get's first index operand
  - The channel symbol itself is erased

This makes `air.channel` of type `gpu_symmetric_heap` syntactic sugar
over cross-rank DMA, with the additional benefit of decoupling the
producer site (where put appears) from the consumer site (where get
appears) via the channel symbol.

- One put and one get per channel symbol
- Both at host scope (no `gpu.launch`/`gpu.func`)
- put's source memref must be `air.symmetric`-tagged
- "Entire memref" form on both sides (no offsets/sizes/strides)
- get must take exactly one index operand (the peer rank)

- `mlir/include/air/Conversion/AIRGpuChannelToMgpuPass.h` — header
- `mlir/include/air/Conversion/GPUPasses.td` — pass def
- `mlir/include/air/Conversion/GPUPassDetail.h` — `GEN_PASS_DEF_AIRGPUCHANNELTOMGPU`
- `mlir/lib/Conversion/AIRGpuChannelToMgpuPass.cpp` — implementation
- `mlir/lib/Conversion/{CMakeLists.txt,Passes.cpp}` — registration
- `mlir/test/Conversion/AIRGpuChannelToMgpu/gpu_channel.mlir` — FileCheck
- `test/gpu/symmetric_heap_dma/air_sym_with_channel.mlir` — high-level e2e
- `test/gpu/symmetric_heap_dma/run.sh` — adds `INPUT=channel` selector

FileCheck unit tests cover:
- Basic put/get pair lowering shape (barrier + mgpuMemcpy with peer-VA)
- Channel symbol is erased after lowering
- Pass is a no-op for non-`gpu_symmetric_heap` channels (e.g., `npu_*`)

End-to-end on rad-mi300a-sh5-1 (SHARE_GPU=1, 2 ranks):
- INPUT=handwritten — PASS
- INPUT=rank — PASS
- INPUT=alloc — PASS
- INPUT=dma — PASS
- INPUT=channel — PASS (chains Phase 6 -> Phase 4 -> Phase 3 -> standard LLVM)
  Both ranks publish their src_buf via channel.put, then read rank 0's slot
  via channel.get. Verification reads back 1.0.

Same SHARE_GPU=1 single-physical-GPU caveat as previous PRs in the stack —
true multi-GPU re-validation is needed before declaring multi-GPU
production-ready.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Add `--multi-gpu` flag to `aircc` that selects the host-only multi-GPU
compilation pipeline:
  1. air-cross-rank-dma-to-mgpu  (Phase 5)
  2. air-gpu-channel-to-mgpu     (Phase 6)
  3. air-symmetric-alloc-to-mgpu (Phase 4)
  4. air-rank-to-mgpu            (Phase 3)
  5. convert-scf-to-cf + convert-to-llvm + reconcile-unrealized-casts

The output is host-only LLVM IR meant to be run as N processes via
`mlir-runner` linked against `libairgpu.so` (and `libmlir_rocm_runtime.so`)
with `RANK` / `WORLD_SIZE` / `LOCAL_RANK` env vars set.

The original Phase 7 plan included a `--multi-rank=N` runner mode that
forks N processes from `aircc` itself. That has been intentionally
deferred: the existing launcher in
`test/gpu/symmetric_heap_dma/run.sh` already does the multi-process
fork+wait pattern in ~30 lines of shell, and wrapping it into `aircc`
adds little value over that. Worth revisiting if real deployment
integration (SLURM, MPI, etc.) becomes a requirement.

- `tools/aircc/aircc.cpp` — adds `--multi-gpu` flag and
  `runMultiGpuCompilation()` function
- `test/gpu/symmetric_heap_dma/run.sh` — adds `INPUT=prelowered SRC=<path>`
  mode that takes `aircc --multi-gpu` output directly
- `docs/MultiGPUPlan.md` — Phase 7 section updated with the new design

- [x] `aircc --target=gpu --multi-gpu` builds and produces clean LLVM IR
  matching the structure of what `INPUT=channel` produces in `run.sh`
- [x] Compiled output uses `llvm.func @mgpuSymmetricHeapInit/Destroy/Get*`,
  `llvm.func @mgpuSymmetricAlloc/Free`, `llvm.func @mgpuMemcpy`,
  `llvm.func @mgpuBarrier`, `llvm.func @mgpuGetHeapBases` (verified
  via `head -20`)
- [ ] E2E run-through of `aircc --multi-gpu` output via `run.sh
  INPUT=prelowered`: deferred — SLURM allocation expired during
  testing. The compile pipeline is byte-for-byte equivalent to the
  manually-invoked `INPUT=channel` pipeline (which we verified PASS),
  so a regression is unlikely.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>